Correcting Contradictions

This paper describes a manual investigation of the contradiction asymmetries of the SICK corpus, which is the intended testing set for a new system for natural language inference. Any system providing conceptual semantics for sentences, so that entailment-contradiction-neutrality relations between sentences can be identified, needs a baseline test set. The investigation of this test set, a part of the SICK corpus, was necessary to check the quality of our testing data and to ensure that the set is logically valid. This checking showed us that the lack of a specific context or reference for the sentence pairs and the presence of indefinite determiners have made the task of annotating very hard, leading to those contradiction asymmetries. We propose a way of correcting these annotations, which solves some of the issues but also makes some compromises.